<html>
<head><meta charset="utf-8"><title>i686-gnu timeouts · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html">i686-gnu timeouts</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="229112443"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229112443" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229112443">(Mar 06 2021 at 17:00)</a>:</h4>
<p><a href="https://github.com/rust-lang/rust/issues/82832">#82832</a> timed out. Today there way at least one other timeout on the i686-gnu job.</p>



<a name="229137488"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229137488" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229137488">(Mar 06 2021 at 23:16)</a>:</h4>
<p>I ran the core tests on i686 in a loop for an hour, and wasn't able to reproduce any hangs. Hopefully it is a CI fluke.  There is a concerning miscompilation in <a href="https://github.com/rust-lang/rust/issues/82833">#82833</a>, but that appears to require a specific collection of flags not used in normal doctests.</p>



<a name="229167981"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229167981" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> tm <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229167981">(Mar 07 2021 at 08:10)</a>:</h4>
<p>It hangs quite consistently <a href="https://github.com/rust-lang/rust/issues/82221">#82221</a>. See also a list of cancelled jobs that took longer than 4h in <a href="https://github.com/rust-lang-ci/rust/actions?query=is%3Acancelled">https://github.com/rust-lang-ci/rust/actions?query=is%3Acancelled</a></p>



<a name="229168162"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229168162" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> tm <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229168162">(Mar 07 2021 at 08:13)</a>:</h4>
<p>I couldn't reproduce this locally neither. On CI, all hangs seems to be tests that spawn rustc. Usually doc tests, but also occasionally UI tests (presuming the issue is the same).</p>



<a name="229206591"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229206591" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229206591">(Mar 07 2021 at 18:13)</a>:</h4>
<p>Thanks @tm for the link. I think I saw that issue when you opened it, but I forgot about it.</p>
<p>I tried running again using Docker, and was able to reproduce:</p>
<div class="codehilite"><pre><span></span><code>test src/char/methods.rs - char::methods::char::escape_debug (line 455) has been running for over 60 seconds
test src/char/methods.rs - char::methods::char::escape_unicode (line 371) has been running for over 60 seconds
test src/char/methods.rs - char::methods::char::from_digit (line 205) has been running for over 60 seconds
test src/char/methods.rs - char::methods::char::from_digit (line 220) has been running for over 60 seconds
test src/char/methods.rs - char::methods::char::from_digit (line 230) has been running for over 60 seconds
test src/char/methods.rs - char::methods::char::is_alphabetic (line 689) has been running for over 60 seconds
test src/char/methods.rs - char::methods::char::is_alphanumeric (line 807) has been running for over 60 seconds
test src/char/methods.rs - char::methods::char::is_ascii (line 1054) has been running for over 60 seconds
test src/char/methods.rs - char::methods::char::is_ascii_alphabetic (line 1213) has been running for over 60 seconds
</code></pre></div>



<a name="229207545"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229207545" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229207545">(Mar 07 2021 at 18:28)</a>:</h4>
<p>All of the rustc processes are hung on reading stdin (on <a href="https://github.com/rust-lang/rust/blob/66ec64ccf31883cd2c28d045912a76179c0c6ed2/compiler/rustc_driver/src/lib.rs#L493">this line</a>).</p>



<a name="229207827"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229207827" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229207827">(Mar 07 2021 at 18:32)</a>:</h4>
<p>Does the container mount all the filesystems it should?</p>



<a name="229207968"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229207968" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229207968">(Mar 07 2021 at 18:34)</a>:</h4>
<p><span class="user-mention" data-user-id="120518">@Eric Huss</span> If you still have the container running, can you check the <code>/proc/$PID/fd</code> directories for those rustc processes to see what fd 0 refers to?</p>



<a name="229208295"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229208295" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229208295">(Mar 07 2021 at 18:39)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span>  I was able to run the tests many times without error, so I assume everything is mounted correctly.  Also, this is an extremely simple docker setup, with very few options.</p>
<p>The fd for stdin of rustc is a pipe to rustdoc: <code>0 -&gt; pipe:[11708854]</code></p>



<a name="229208376"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229208376" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229208376">(Mar 07 2021 at 18:40)</a>:</h4>
<p>Can you check lsof to figure out who has the other end open?</p>



<a name="229208405"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229208405" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229208405">(Mar 07 2021 at 18:41)</a>:</h4>
<p>Yea, rustdoc is hung in spawn:</p>
<div class="codehilite"><pre><span></span><code>#0  0xf7f70569 in __kernel_vsyscall ()
#1  0xeee29be2 in __lll_lock_wait () at ../sysdeps/unix/sysv/linux/i386/lowlevellock.S:144
#2  0xeee26571 in __GI___pthread_rwlock_unlock (rwlock=0xef01a124 &lt;_ZN3std3sys4unix2os8ENV_LOCK17h937e2955b3dbd3d2E.llvm.373065096730365974&gt;)
    at pthread_rwlock_unlock.c:42
#3  0xeeee3b4f in std::sys::unix::process::process_inner::_$LT$impl$u20$std..sys..unix..process..process_common..Command$GT$::spawn::h61a7f768f84ef457 ()
   from /checkout/obj/build/i686-unknown-linux-gnu/stage1/bin/../lib/libstd-901ad3224252f4d9.so
</code></pre></div>



<a name="229208419"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229208419" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229208419">(Mar 07 2021 at 18:41)</a>:</h4>
<p>If it's hanging, that suggests that someone has the pipe write end open and hasn't closed it.</p>



<a name="229208422"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229208422" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229208422">(Mar 07 2021 at 18:41)</a>:</h4>
<p>No line numbers, but my first guess is that rwlock is the env_lock.</p>



<a name="229208427"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229208427" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229208427">(Mar 07 2021 at 18:41)</a>:</h4>
<p>Interesting!</p>



<a name="229208439"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229208439" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229208439">(Mar 07 2021 at 18:42)</a>:</h4>
<p>The unix spawn code is a little tricky, in the way it handles locks around <code>fork()</code>.</p>



<a name="229208538"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229208538" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229208538">(Mar 07 2021 at 18:43)</a>:</h4>
<p>Oh, to be clear, there are two rustdoc processes.</p>
<p>The second one has a single thread, which is blocked on that call to unlock.  So I assume that is just after the call to <code>fork</code>.</p>



<a name="229209238"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229209238" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229209238">(Mar 07 2021 at 18:51)</a>:</h4>
<p>That's very strange. I'm wondering why it can't just snapshot the environment and set it post-fork.</p>



<a name="229209488"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229209488" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229209488">(Mar 07 2021 at 18:55)</a>:</h4>
<p>There is some pretty gnarly interaction of posix rwlock and threads and forking.  I ran across <a href="https://stackoverflow.com/questions/61976745/why-does-rust-rwlock-behave-unexpectedly-with-fork">https://stackoverflow.com/questions/61976745/why-does-rust-rwlock-behave-unexpectedly-with-fork</a> which explains some of it.</p>



<a name="229209538"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229209538" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229209538">(Mar 07 2021 at 18:56)</a>:</h4>
<p>I notice that it is not using <code>posix_spawn</code>, and instead is using fork/exec. I wonder why...</p>



<a name="229209662"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229209662" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229209662">(Mar 07 2021 at 18:58)</a>:</h4>
<p>And yeah, dropping env_read_lock in the child when it wasn't taken in the child seems like a bug, unless env_read_lock is something really unusual.</p>



<a name="229209667"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229209667" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229209667">(Mar 07 2021 at 18:59)</a>:</h4>
<p>Ah, glibc is 2.23, and it requires &gt;=2.24.</p>



<a name="229209799"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229209799" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229209799">(Mar 07 2021 at 19:00)</a>:</h4>
<p>Which it doesn't appear to be...</p>



<a name="229209944"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229209944" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229209944">(Mar 07 2021 at 19:02)</a>:</h4>
<p>This is still a bug in the case where fork is used, even if using posix_spawn would work around it for rustdoc.</p>



<a name="229210126"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229210126" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229210126">(Mar 07 2021 at 19:05)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> If I'm reading this correctly, it is wrong for the child process to call <code>pthread_rwlock_unlock</code> because its thread number will be different from where it was acquired, right?  Like, it should only call <code>pthread_rwlock_unlock</code> in the parent?</p>



<a name="229210191"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229210191" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229210191">(Mar 07 2021 at 19:06)</a>:</h4>
<p>Right, but the lock may still appear held in the child.</p>



<a name="229210199"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229210199" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229210199">(Mar 07 2021 at 19:06)</a>:</h4>
<p>It just can't be unlocked safely.</p>



<a name="229210899"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229210899" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229210899">(Mar 07 2021 at 19:16)</a>:</h4>
<p>Oh, this is a <em>recent</em> change.</p>



<a name="229210927"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229210927" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229210927">(Mar 07 2021 at 19:16)</a>:</h4>
<p>Commit <a href="https://github.com/rust-lang/rust/commit/55ca27faa78434d81d3f59535c96bbb2a08f0d5c">55ca27faa78434d81d3f59535c96bbb2a08f0d5c</a> last month switched this to an rwlock.</p>



<a name="229210950"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229210950" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229210950">(Mar 07 2021 at 19:17)</a>:</h4>
<p>I'm guessing that we didn't have this CI issue before that commit.</p>



<a name="229211028"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211028" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211028">(Mar 07 2021 at 19:18)</a>:</h4>
<p><a href="https://github.com/rust-lang/rust/pull/81850">https://github.com/rust-lang/rust/pull/81850</a></p>



<a name="229211167"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211167" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211167">(Mar 07 2021 at 19:20)</a>:</h4>
<p>Yeah, that makes sense. Unlocking a mutex post-fork probably works (though I don't know if it's defined behavior or not).</p>



<a name="229211192"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211192" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211192">(Mar 07 2021 at 19:20)</a>:</h4>
<p>Unlocking an rwlock post-fork doesn't, because of the thread-ID.</p>



<a name="229211318"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211318" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211318">(Mar 07 2021 at 19:22)</a>:</h4>
<p>There may be ways to improve this further, but for the moment, I think the right approach may be to revert that series of commits (including on stable if it's been in a stable release).</p>



<a name="229211457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Yuki Okushi <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211457">(Mar 07 2021 at 19:25)</a>:</h4>
<p><span class="user-mention silent" data-user-id="239881">Josh Triplett</span> <a href="#narrow/stream/242791-t-infra/topic/i686-gnu.20timeouts/near/229211318">said</a>:</p>
<blockquote>
<p>There may be ways to improve this further, but for the moment, I think the right approach may be to revert that series of commits (including on stable if it's been in a stable release).</p>
</blockquote>
<p>fortunately, it's on nightly (according to the milestone)</p>



<a name="229211462"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211462" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211462">(Mar 07 2021 at 19:25)</a>:</h4>
<p>Ah, good.</p>



<a name="229211465"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211465" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211465">(Mar 07 2021 at 19:25)</a>:</h4>
<p><span class="user-mention" data-user-id="120518">@Eric Huss</span> Do you want to prepare a revert and I'll review, or vice versa?</p>



<a name="229211564"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211564" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211564">(Mar 07 2021 at 19:27)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> Sure</p>



<a name="229211572"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211572" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211572">(Mar 07 2021 at 19:27)</a>:</h4>
<p>(which way around?)</p>



<a name="229211653"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211653" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211653">(Mar 07 2021 at 19:28)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> haha, I'll post a PR soon.</p>



<a name="229211655"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211655" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211655">(Mar 07 2021 at 19:28)</a>:</h4>
<p>Alright. I can write a commit message if that helps:</p>



<a name="229211677"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211677" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211677">(Mar 07 2021 at 19:29)</a>:</h4>
<p>I want to run a stress test first just to ensure this actually solves the problem.</p>



<a name="229211684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229211684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229211684">(Mar 07 2021 at 19:29)</a>:</h4>
<p><span aria-label="+1" class="emoji emoji-1f44d" role="img" title="+1">:+1:</span></p>



<a name="229212283"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229212283" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229212283">(Mar 07 2021 at 19:38)</a>:</h4>
<p>Commit message:</p>
<p>Revert switch of env locking to rwlock, to fix deadlock in process spawning</p>
<p>PR <a href="https://github.com/rust-lang/rust/pull/81850">https://github.com/rust-lang/rust/pull/81850</a> switched the environment lock from a mutex to an rwlock. However, process spawning (when not able to use posix_spawn) locks the environment before forking, and unlocks it after forking (in both the parent and the child). With a mutex, this works. With an rwlock, on at least some targets, unlocking in the child does not work correctly, resulting in a deadlock.</p>
<p>This has manifested as CI hangs on i686 Linux; that target doesn't use posix_spawn in the CI environment due to the age of the installed C library. (Switching to posix_spawn would just mask this issue, though, which would still arise in any case that can't use posix_spawn.)</p>
<p>Some additional cleanup of environment handling around process spawning may help, but for now, revert the PR and go back to a standard mutex.</p>



<a name="229215509"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229215509" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> tm <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229215509">(Mar 07 2021 at 20:28)</a>:</h4>
<p>While neither rwlock nor mutex is correct here (corresponding functions are not async-signal-safe, so cannot be used after fork), I wonder what breaks exactly. The rwlock stores a thread id of a writer, while this code obtains read lock.</p>



<a name="229217453"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229217453" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> tm <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229217453">(Mar 07 2021 at 21:00)</a>:</h4>
<p>In 2.23, to unlock an rwlock the implementation obtains an internal lock. It is possible that another thread obtained the lock at fork time, so it will remain locked. The implementation looks quite differently now, which is probably why I couldn't reproduce this.</p>



<a name="229217727"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229217727" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229217727">(Mar 07 2021 at 21:04)</a>:</h4>
<p>Yea, I was just taking a look, and the lll_lock code is in assembly in that version.</p>



<a name="229218304"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229218304" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229218304">(Mar 07 2021 at 21:13)</a>:</h4>
<p>Posted <a href="https://github.com/rust-lang/rust/issues/82877">#82877</a></p>



<a name="229219084"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229219084" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229219084">(Mar 07 2021 at 21:25)</a>:</h4>
<p><span class="user-mention" data-user-id="120518">@Eric Huss</span> LGTM, r+ed.</p>



<a name="229225574"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229225574" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Notification Bot <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229225574">(Mar 07 2021 at 23:10)</a>:</h4>
<p>This topic was moved by <span class="user-mention silent" data-user-id="239881">Josh Triplett</span> to <a class="stream-topic" data-stream-id="219381" href="/#narrow/stream/219381-t-libs/topic/environment.2C.20locking.2C.20fork.2C.20and.20Command">#t-libs &gt; environment, locking, fork, and Command</a></p>



<a name="229344908"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229344908" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229344908">(Mar 08 2021 at 17:58)</a>:</h4>
<p>Getting spurious CI failures trying to merge the commit that'll fix unrelated spurious CI failures...</p>



<a name="229344929"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229344929" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229344929">(Mar 08 2021 at 17:59)</a>:</h4>
<p><a href="/user_uploads/4715/pBJx1WKVJ05l2JtLM6m_mwWX/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/pBJx1WKVJ05l2JtLM6m_mwWX/image.png" title="image.png"><img src="/user_uploads/4715/pBJx1WKVJ05l2JtLM6m_mwWX/image.png"></a></div>



<a name="229358166"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/i686-gnu%20timeouts/near/229358166" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/i686-gnu.20timeouts.html#229358166">(Mar 08 2021 at 19:19)</a>:</h4>
<p>Maybe attempt <a href="https://github.com/rust-lang/rust/issues/6">#6</a> will be more lucky. <span aria-label="stuck out tongue wink" class="emoji emoji-1f61c" role="img" title="stuck out tongue wink">:stuck_out_tongue_wink:</span></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>